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SCHEDULING THE DISPATCH OF CELLS IN MULTISTAGE SWITCHES 
§ 0. PRIORITY CLAIM 

Benefit is claimed, under 35 U.S.C. § 119(e)(1), 
to the filing date of: provisional patent application 
serial number 60/252,006, entitled "CRRD: A CONCURRENT 
ROUND-ROBIN DISPATCHING SCHEME FOR CLOS-NETWORK SWITCHES", 
filed on November 20, 2000 and listing Jonathan Chao and 
Eiji Oki as the inventors; and provisional patent 
application serial number 60/253,335, entitled ''A SCALABLE 
ROUND-ROBIN BASED DISPATCHING SCHEME FOR LARGE-SCALE 
CLOS-NETWORK SWITCHES", filed on November 27, 2000 and 
listing Jonathan Chao and Eiji Oki as inventors, for any 
inventions disclosed in the manner provided by 35 U.S.C. 
§ 112, % 1. This provisional application is expressly 
incorporated herein by reference. 

§ 1. BACKGROUND 

§ 1.1 FIELD OF THE INVENTION 

The present invention concerns the communication 
of data over networks, such as the Internet for example. 
More specifically, the present invention concerns 
scheduling the servicing (e.g., dispatching) of cells or 
packets buffered at input ports of a switch, 

§ 1.2 RELATED ART 

Switches and routers are used in networks, such as the 
Internet for example, to forward data towards its destination. 

1 

Express Mail No. EL667251 302US 



PolY-17/APP 

The need for large-scale switches and routers is introduced in 
§ 1.2.1 below. Then, two types of switches, as well as 
disadvantages and challenges in each of these two types of 
switches, are introduced in § 1.2.2 below. 

5 

§ 1.2.1 THE NEED FOR LARGE-SCALE (e.g., 
TERABIT) ROUTERS AND SWITCHES 

Many expect that Internet traffic will continue to 
10 grow explosively. Given this assumption, high-speed switches 
and routers (e.g., those having a throughput over one Terabit 
Q per second) will become necessary. Most high-speed packet 
^J- switches adopt a fixed- size cell in the switch fabric. If 
111 variable length packets are to be supported in the network, such 
j:;15 packets may be segmented and/or padded into fixed-sized cells 
upon arrival, switched through the fabric of the switch^ and 
reassembled into packets before departure. There are two main 
types of high-speed switches, each introduced in § 1.2.2 below. 

§ 1.2.2 TYPES OF SWITCHES (SINGLE STAGE VERSUS 
P MULTI-STAGE) AND THEIR CHARACTERISTICS 

There are two main types of high-speed switches -- 
namely single stage and multi-stage. Single stage switches and 
25 perceived limits of single stage switches are introduced in §§ 
1.2.2.1 and 1.2.2.2, respectively, below. Similarly, 
multi-stage switches and perceived limits of multi-stage 
switches are introduced in §§ 1.2.2.3 and 1.2.2.4, respectively, 
below. 

30 
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§ 1.2.2.1 SINGLE STAGE SWITCH ARCHITECTURES 

A so-called ''crossbar'' switch is a classic example of 
a single stage switch. In a crossbar switch, identical 
5 switching elements are arranged on a matrix plane. The article, 
N. McKeown, M. I22ard, A. Mekkiltikul, W. Ellerisick and M. 
Horowitz, "Tiny-Tera: A Packet Switch Core,'' IEEE Micro . , pp. 
26-33, (Jan. -Feb. 1997) (incorporated herein by reference and 
hereafter referred to as ''the Tiny-Tera article") proposes a 320 
10 gigabit per second crossbar switch fabric that uses a 

centralized scheduler referred to as "iSLIP" . The article H. J. 
Chao and J-S Park, "Centralized Contention Resolution Schemes 
[Jt for a Large-Capacity Optical ATM Switch," Proc. IEEE ATM 
^Z, Workshop '97 (Fairfax, VA, May 1998) (incorporated herein by 
P15 reference and hereafter referred to as "the Chao article") 

introduced a centralized contention resolution scheme for a 
Li large capacity crossbar optical switch. The article E. Oki, N. 

p Yamanaka, Y. Ohtomo, K. Okazaki and R. Kawano, "A 10-Gb/s (1.25 

J^; Gb/s X 8) 4 x 2 0.25 Mm CMOS/SIMOX ATM Switch Based on Scalable 

20 Distributed Arbitration, " IEEE J. of Solid-State Circuits , Vol. 
34, No. 12, pp. 1921-1934 (Dec. 1999) (incorporated herein by 
reference and hereafter referred to as "the Oki article") 
describes a high-speed switch chip having a throughput of 40 
Gb/s, for a high-speed crosspoint-buf f ered crossbar switching 
25 system. 

§ 1.2.2.2 LIMITS OF SINGLE STAGE SWITCH 
ARCHITECTURES 

30 The switching techniques used in single stage 

switches, such as those described in the articles referenced in 
§ 1,2.2.1, are effective up to a certain switch size. 
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Unfortunately, however, with such techniques, the complexity of 
the switching elements is proportional to the square of the 
number of switch ports. As a practical matter, this limits the 
feasibility of using such techniques in a large scale (e.g., 
5 Terabit per second and above) switch. 

§ 1.2.2.3 MULTI-STAGE SWITCH ARCHITECTURES 

In view of the limits of single stage switches, 
10 multiple- stage switch architectures have been introduced. For 

example, the so called ''Clos -network'' switch has three stages 
^0 and is very attractive because of its scalability. See the 

ifl article, C. Clos, ''A Study of Non-Blocking Switching Networks,'' 

Bell Sys. Tech. Jour. , pp. 406-424 (March 1953) (incorporated 
ffl 15 herein by reference and hereafter referred to as ''the Clos 

article") . More specifically, the three stages include (i) 
Qj input modules, (ii) a central switching fabric (incorporated in 

rj central modules), and (iii) output modules. 

20 Clos-network switches have been categorized into two 

types those with buffers to store cells in the second-stage 
(central) modules and those with no buffers in the second-stage 
(central) modules. For example, the article, T. Chaney, J. A. 
Fingerhut, M. Flucke, J. S, Turner, ''Design of a Gigabit ATM 

25 Switch," Proc. IEEE INFOCOM '97 , pp. 2-11 (April 1997) 

(incorporated herein by reference and hereafter referred to as 
''the Chaney article") discusses a gigabit ATM switch using 
buffers in the second-stage modules. In the switch architecture 
discussed in the Chaney article (hereafter referred to as "the 

30 Chaney switch"), every incoming cell is randomly distributed 

from the first-stage to the second-stage module to balance the 
traffic load in the second-stage. This is commonly referred to 
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as ^'random dispatching'' . The buffers in the second-stage 
modules are used to resolve contention among cells from 
different first-stage modules. See the article J. Turner and N. 
Yamanaka, "Architectural Choices in Large Scale ATM Switches," 
5 lEICE Trans, Commun, , Vol. E81-B, No. 2, pp. 120-137 (Feb. 1998) 
(incorporated herein by reference and hereafter referred to as 
''the Turner article") . 

An example of a switch architecture with no buffers in 
10 the second- stage modules is provided in the article P. M, 
Chiussi, J. G. Kneuer, and V. P. Kumar, ''Low-Cost Scalable 
\Q Switching Solutions for Broadband Networking: The ATLANTA 
JtJ Architecture and Chipset," IEEE Commun . Mag . , pp. 44-53 {Dec. 
H; 1997) (incorporated herein by reference and hereafter referred 
g|15 to as ''the Chiussi article'') . Since there are no buffers in the 

second-stage modules to resolve potential contention, how cells 
O are dispatched from the first-stage to the second-stage becomes 
fi important. The simple random distribution used in the Chaney 
•J; switch may be used in the switch discussed in the Chiussi 
y.20 article (hereafter referred to as "the Chiussi switch") under 
certain conditions. Given the potential for contention, some 
buffers are provided in the first and third stages of the 
Chiussi switch. Although there are some studies of routing 
algorithms where every stage has no buffers, such algorithms 
25 require a contention resolution function for output ports, 

before cells even enter the multiple-stage switches. See, e.g., 
the articles, C. Y. Lee and A. Y. Qruc, "A Fast Parallel 
Algorithm for Routing Unicast Assignments in Benes Networks," 
IEEE Trans, on Parallel and Distributed Sys . , Vol. 6, No. 3, pp. 
30 329-333 (March 1995), and T. T. Lee and S-Y Liew, "Parallel 

Routing Algorithms in Benes-Clos Networks," Proc. IEEE INFQCQM 
' 96 , pp. 279-286 (1996) . (Both of these articles are 
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incorporated herein by reference.) Such a pre-switch contention 
resolution function is challenging to implement in high-speed 
switches . 

5 § 1,2.2.4 PROBLEMS WITH KNOWN MULTIPLE- STAGE 

SWITCH ARCHITECTURES 

To provide high performance (e.g., high throughput), 
known multiple-stage switch architectures will typically require 
10 speed-up of the internal switching fabric (i.e., in the 

second- stage modules) and/or resequencing . For example, the 
1:3 Chaney switch provides high performance if the internal speed-up 
li? factor is set to be more than 1.25. As is known, speed-up of 
IJ1 the switching fabric can be accomplished by increasing the 
Vl5 number of central modules and/or increasing the internal link 
"f' speed used in the central modules. Unfortunately, however, in 
I- the Chaney switch, buffers used in the second-stage modules 

Jfi cause cells to become out-of -sequence . Since the Chiussi switch 
Q does not use buffers in its second-stage modules, cell 
f:-20 out-of -sequence problems are not encountered. Assuming that a 
random dispatching scheme is used to forward cells from the 
first stage to the second stage of the Chiussi switch, to 
minimize the chance of contention at the second stage, the 
internal speed-up factor has to be increased further. For 
25 example, to achieve 100 % throughput using random dispatching in 
a large-size Chiussi switch, the speed-up is set to about 1.6 as 
indicated in the Chiussi article. 

As should be appreciated from the foregoing, to be 
30 used in today's more demanding environments, the Chaney switch 
requires cell resequencing, which is challenging, and at least 
moderate speed-up, which increases costs. To be used in today's 
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more demanding environments, the Chiussi switch requires even 
more speedup, which increases costs even more. In view of these 
problems with known switch architectures, a scalable switch 
architecture having a high throughput, avoiding out -of -sequence 
5 cells (e.g., by eliminating buffers in the second stage), and 
avoiding speed-up of the switch fabric is coveted. 

§ 2. SUMMARY OF THE INVENTION 

10 The present invention may be used to provide a 

scalable switch architecture that has a high throughput, 
i|j avoids out -of -sequence cells, and avoids speedup of the 

switch fabric is coveted. The present invention may do so 
u^. by providing a cell dispatch scheduling method for use in a 
^fi;15 multi-stage switch including a plurality of central 

modules, and a plurality of input modules, each including 
P virtual output queues and outgoing links coupled with each 

of the plurality of central modules. In one embodiment of 
m the cell dispatch scheduling method, a non-empty virtual 
y,20 output queue of an input module is matched with an outgoing 

link in the input module^ and the outgoing link of the 

input module is matched with an outgoing link of one of the 

central modules . 

25 In one embodiment, a non-empty virtual output 

queue of an input module may be matched with an outgoing 
link in the input module by (i) broadcasting a request for 
the non-empty virtual output queue to an arbiter for each 
of the outgoing links of the input module, (ii) selecting, 

30 with the arbiter of each of the outgoing links of the input 
module, a non-empty virtual output queue that broadcast a 
request, (iii) sending a grant to an arbiter for the 
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selected non-empty virtual output queue, and (iv) 
selecting, with the arbiter of the selected non-empty 
virtual output queue, an outgoing link from among the one 
or more outgoing links that sent a grant. In one version 
5 of such an embodiment, the act of matching a non-empty 
virtual output queue of an input module with an outgoing 
link in the input module may occur within one cell time 
slot. In one version of such an embodiment, the act of 
selecting, with the arbiter of each of the outgoing links 
10 of the input module, a non-empty virtual output queue that 
broadcast a request, may be done in accordance with a round 
I|I robin discipline. The round robin discipline may be 
5 effected by a pointer. In one embodiment, the pointer may 

move through groups of virtual output queues, before moving 
!5l5 through virtual output queues within each group. 

PI In one embodiment, the acts of (i) broadcasting a 

request for the non-empty virtual output queue to an 
m arbiter for each of the outgoing links of the input module, 
y20 (ii) selecting, with the arbiter of each of the outgoing 

links of the input module, a non-empty virtual output queue 
that broadcast a request, (iii) sending a grant to an 
arbiter for the selected non-empty virtual output queue, 
and (iv) selecting, with the arbiter of the selected 
25 non-empty virtual output queue, an outgoing link from among 
the one or more outgoing links that sent a grant, may be 
performed at least twice within one cell time slot. 

In one embodiment, the act of matching the 
30 outgoing link of the input module with an outgoing link of 
one of the central modules may be performed by (i) 
broadcasting a request for the outgoing link of the input 
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module to an arbiter for each of the outgoing links of the 
central modules that lead towards an output port associated 
with the virtual output queue matched with the outgoing 
link of the input module, (ii) selecting with the arbiter 
5 of each of the outgoing links of the central modules, an 
outgoing link of an input module that broadcast a request; 
and (iii) sending a grant to the selected outgoing link of 
the input module. 

10 The teachings of the present invention may be 

used for matching a non-empty virtual output queue of an 
input module with an outgoing link in the input module. 



The present invention also provides a teaching of 
yillS apparatus for effecting the various methods. The present 

invention also provides a teaching of various data 
D structures that may be used for effecting the various 
r% methods . 

M 20 § 3. BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram of an exemplary 
multiple-stage switch in which the present invention may be 
implemented. 

25 

Figure 2 is a bubble diagram of operations that 
may be performed by the present invention. 

Figure 3 is a flow diagram illustrating an 
30 exemplary virtual output queue method that may be performed 
under the present invention. 
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Figure 4 is a flow diagram illustrating an 
exemplary link arbitration method that may be performed 
under the present invention. 

5 Figure 5 is a flow diagram illustrating an 

exemplary virtual output queue arbitration method that may 
be performed under the present invention. 

Figure 6 is a flow diagram illustrating an 
10 exemplary outgoing link method that may be performed under 
the present invention. 

Figure 7 is a flow diagram illustrating an 
exemplary central module arbitration method that may be 
Sl5 performed under the present invention. 



Figure 8 is a block diagram of components of an 

exemplary apparatus, as well as inter- component signaling 

links, that may be used to perform operations under the 
20 present invention. 



Figures 9(a) through 9(g) illustrate an example 
of how a first phase of an exemplary dispatching method 
under the present invention matches non-empty virtual 
25 output queues to outgoing links in an input module. 

Figure 10 illustrates exemplary data structures 
that may be used to store state information for use by a 
link arbitration operation. 



30 



Figure 11 illustrates exemplary data structures 
that may be used to store state information for use by a 
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VOQ arbitration operation. 

Figure 12 illustrates exemplary data structures 
that may be used to store state information for use by- a 
5 central module arbitration operation. 

§ 4. DETAILED DESCRIPTION OF THE INVENTION 

The present invention involves novel methods, 
10 apparatus and data structures for dispatching cells or 

packets buffered at the input port of a multi-stage switch. 
The following description is presented to enable one 
skilled in the art to make and use the invention, and is 
provided in the context of particular embodiments and 
15 methods. Various modifications to the disclosed 

embodiments and methods will be apparent to those skilled 
in the art, and the general principles set forth below may 
be applied to other embodiments, methods and applications. 
Thus, the present invention is not intended to be limited 
20 to the embodiments and methods shown and the inventors 

regard their invention as the following disclosed methods, 
apparatus and materials and any other patentable subject 
matter to the extent that they are patentable. 

25 § 4.1 EXEMPLARY ENVIRONMENT 

The present invention may be practiced in a 
multiple-stage switch, such as a Clos-network switch for 
example. Figure 1 is a high-level block diagram of a 
30 Clos-network switch 100. The switch 100 basically includes 
input modules (^^IMs") 12 0 at a first stage, central modules 
(''CMS") 14 0 at a second stage, and output modules C'OMs") 
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160 at a third stage. 

As shown in Figure 1, each input module (IM) 12 0 
includes a number (n) of input ports (^^IPs") 110. Thus, if 
5 there are a number (k) of input modules (IMs) 12 0, there 
will be a total of n*k input ports (IPs) 110. Similarly, 
each output module (OM) 160 includes a number (n) of output 
ports (""OPs") 170. Thus, if there are a number (k) of 
output modules (OMs) 160, there will be a total of n*k 
10 output ports (OPs) 170. Each output port (OP) 170 may 
include a buffer 165 receiving, at most, m cells in one 
'i cell time slot. The buffer size should be large enough to 

avoid cell loss. Each output port (OP) 170 forwards one 
U cell in a first-in-first-out (FIFO) manner, to an 
Jtl5 associated output line. 

P=n; A number (m) of central modules (CMs) 140 are 

T arranged between the input modules (IMs) 12 0 and output 
til modules (OMs) 160. More specifically, as shown, each input 
!::20 module (IM) 12 0 includes a number (m) of outgoing links Li 
13 0, each connecting the input module (IM) 12 0 to a 
different one of the m central modules (CMs) 140. 
Similarly, each central module (CM) 140 includes a number 
(k) of outgoing links Lo 150, each connecting the central 
25 module (CM) 14 0 with a different one of the k output 
modules (OMs) 160. 

Finally, each of the input modules (IM) 12 0 may 
include a number of virtual output queues (VOQs) 12 5, each 
30 of the VOQs 125 being associated with an output port (OP) 

170. The VOQs 125 are used to eliminate problems caused by 
head-of-line (^^HOL") blocking. A given VOQ 125 can receive 
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at most n cells from n input ports (IPs) 110, and can send 
one cell to a central module (CM) 140 in one cell time 
slot . 

The following terminology: 

n ^ the number of input ports and output ports for 

each IM and OM, respectively; 
k = the number of IMs, as well as the number of OMs; 
m ^ the number of CMs; 

IM(i) = the ith input module, where 0 < i < k-l; 
CM(r) = the rth central module, where 0 < r < m-l; 
OM(j) = the jth output module, where 0 < j < k-1; 
IP(i,h) = the hth input port at IM(i), where 0 < h 
< n-1; 

OP(j^h) ^ the h^^ output port at OM(j), where 0 < h < 
n-1; 

VOQ(i,j,h) - the VOQ in IM(i) that stores cells 

destined for OP(j,h); 
Li(i,r) = the link between IM(i) and Cn{r) ; and 
Lo(r,j) = the link between CM(r) and OM{j) , 

may be used in the specification that follows. 

The first stage of the switch 100 may include k 
input modules (IMs) 12 0, each of which has an n-by-m 
dimension. The second stage of the switch 10 0 may include 
m central modules (CMs) 140, each of which has a k-by-k 
dimension. The central modules (CMs) 140 are preferably 
buffer-less, thereby avoiding the cell out-of -sequence 
problems introduced in § 1.2.2.4 above. The third stage of 
the switch 100 may include k output modules (OMs) 160, each 
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of which has an m-by-n dimension. 

§ 4.2 FUNCTIONS THAT MAY BE PERFORMED 

5 A first aspect of the present invention may 

function to provide a scalable multiple-stage switch, able 
to operate at high throughput, without needing to resort to 
speeding up the switching fabric and without needing to use 
buffers in the second stage. Accordingly, the present 
10 invention may avoid the cost of speed-up and the cell 

out -of -sequence problems that may occur when buffers are 
used in the second stage. The present invention may do so 
D using a multiple phase cell dispatch scheme, each phase 
U using a simple and fair (e.g., round robin) arbitration 
;fc;i5 methods. More specifically, the present invention may 

function to provide a multiple phase cell dispatch scheme 
Pi; in which VOQs of an input module and outgoing links of the 
2 input module are matched in a first phase, and in which an 
m outgoing link of an input module is matched with an 
rhO outgoing link of a central module in a second phase. The 
arbiters become de synchronized under stable conditions 
which contributes to' the switch's high throughput 
characteristic . 

25 The present invention may also function to relax 

a dispatch scheduling time and reduce the complexity of 
interconnections between arbiters. The present invention 
may do so by arranging output link arbiters as master and 
slave arbiters, operated in a hierarchical manner. More 

30 specifically, the VOQs of an input module may be arranged 
into groups. For each outgoing link of the input module, a 
master arbiter may select a group of VOQs from among a 
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number of candidate groups, and a slave arbiter may select 
a VOQ from among the VOQs belonging to the selected group. 

Finally, the present invention may function to 
5 relax the time (e.g., from less than one cell time slot to 
more than one cell time slot) needed to schedule a cell 
dispatch. The present invention may do so by introducing 
more than one subscheduler , each of which is allowed to 
take more than one time slot for dispatching, although one 
10 of the subschedulers provides a dispatching result within 
each cell time slot. 

§ 4.3 EXEMPLARY OPERATIONS 

15 Figure 2 is a bubble chart illustrating 

operations that may be performed in accordance with the 
present invention. When a cell 205 arrives at a VOQ, a VOQ 
(non-arbitration) operation 210 may broadcast (as indicated 
by ellipses) a request 215 to a plurality of link 

20 arbitration operations 230, each associated with a given 
outgoing link 130 of the input module (IM) 120. The 
request 215 is indicated as a first communication C'l"). 

Based on link arbitration state information (e.g. 

25 a round-robin pointer) 235, in each case, the link 

arbitration operations 23 0 will select one of the candidate 
VOQs that submitted a request 215. It 230 will then send a 
grant 237 to an arbitration operation 240 of the selected 
VOQ. The grant 23 7 is indicated as a second communication 

30 (''2") . The link arbitration operation 230 may also send 
declines (not shown) to non- selected VOQs, though such a 
communication (s) is not necessary. 



15 



Poly-17/APP 

Since the VOQ operation 210 broadcasts a request 
(e.g., to all the outgoing links of the input module), it's 
corresponding arbitration operation 240 may receive more 
5 than one grant (i.e., one grant each from more than one 

outgoing link arbitration operation 230) . Accordingly, the 
VOQ arbitration operation 24 0 uses VOQ arbitration state 
information (e.g., a round robin pointer) 245 to select one 
of the outgoing links from those candidates providing a 
10 grant. The VOQ arbitration operation 240 then communicates 
a grant 247 to link operations 250 associated with the 
selected link. This grant is indicated as third 
communication (''3") . 



;5:15 At this point, a cell at a VOQ may have been 

M= (i.e., if it won an arbitration) matched with an outgoing 

p. link 130 of the input module (IM) 120. These operations 

[f; related to generating such a VOQ-Li match may be referred to 

m as the first phase of the cell dispatch scheduling 

rf20 invention. This first phase can be summarized as two 

steps. In the first step, at each time slot, non-empty 
VOQs send (e.g., multicast, or preferably broadcast) 
requests to use any one of a number of outgoing links Li (of 
an input module (IM) ) in a next cell time slot. In the 
25 second step, each outgoing link Li sends a grant to a 

selected VOQ. If a given VOQ receives more than one grant, 
it may select one of the outgoing links from which it 
received a grant. Accordingly, the input module (IM) 
selects at most m request out of n^k non-empty VOQs. If 
30 these steps can be repeated within a cell time slot, 

multiple iterations can be run such that non-empty VOQs 
failing to be matched with an available outgoing link may 
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retry during a next iteration. If multiple iterations are 
run, the VOQs and/or outgoing links Li may maintain state 
information related to whether they are available or 
reserved for the next cell time slot. 

5 

Still referring to Figure 2, a link operation 250 
associated with each outgoing link 130 matched with a VOQ, 
may submit a request 255 to a central module (CM) 
arbitration operation 260, The central module (CM) is 
10 associated with the outgoing link (Li) . Each central module 
will have k arbiters, each corresponding to an output 
% module (OM) . The arbiter serving the request will 
W correspond with the output module (OM) defined by the 
M= non-empty VOQ that is matched with the outgoing link Li that 
;;;:15 originated the request. This request 255 is indicated as a 
fourth communication ("M") . Alternatively, the VOQ 
selecting an outgoing link can originate such a request 
1} (4M as indicated by dashed line 290. 

rf'20 Each CM arbitration operation 2 60 may select one 

of the candidate requesting outgoing links 13 0 based on 
IM-CM arbitration state information 265. It 260 then 
communications a grant 267 back to the link operation 250 
associated with the selected one of the candidate 
25 requesting output links 130. This grant 267 is indicated 
as a fifth communication ('^5''). 

The link operation 25 0 may then send an 
indication 270 to the VOQ operation 210 that it may send 
30 (i.e., dispatch) its head-of-line (^^HOL'O cell. This 

indication 270 is indicated as a sixth communication (''6") . 
The various arbitration operations 230, 240, 260 may update 
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their state information 235, 245, 255. Further, other 
state information 212 may be updated by its associated 
operation 210 . 

5 Various alternatives of when state information is 

updated will be apparent to those skilled in the art. For 
example, each of the outgoing link arbitration operation (s) 
230 may employ a pointer (to VOQ identifiers) , updated in a 
round- robin manner. The pointer may be updated (a) when a 
10 VOQ request is granted, (b) when a VOQ grant is received, 
or, preferably (c) when a central module grant is received. 

■S In servicing VOQs, it is possible for the arbitration 

Si operation 230 to pass over empty (i.e., non- requesting) 

VOQs. Therefore, when the pointer is updated, it can (a) 

iSl5 go to a next VOQ from the VOQ pointed to at the start of 
the cell time slot, or (b) go to a next VOQ from the VOQ 

Of selected. 



m Similarly, each of the VOQ arbitration 

y20 operation(s) 240 may employ a pointer (to outgoing links), 
updated in a round-robin manner. The pointer may be 
updated (a) when a grant is received from an outgoing link 
Li, (b) when an outgoing link is selected by the VOQ 
arbitration operation, or (c) when a central module grant 
25 is indicated. It is possible for the arbitration operation 
240 to pass over non-granting outgoing links Li. Therefore, 
when the pointer is updated, it can (a) go to a next 
outgoing link Li from the outgoing link Li pointed to at the 
start of the cell time slot, or (b) go to a next outgoing 
30 link Li from the selected outgoing link Li. 
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Finally, each of the CM arbitration operation (s) 
260 may employ a pointer (to outgoing links Li) , updated in 
a round-robin manner. The pointer may be updated upon a 
grant by the CM arbitration operation 260. In servicing 
5 outgoing links, it is possible for the CM arbitration 

operation to 2 60 to pass over non- requesting outgoing links 
Li, Therefore, when the pointer is updated, it can (a) go 
to a next outgoing link Li from the outgoing link Li pointed 
to at the start of the cell time slot, or (b) go to a next 
10 outgoing link Li from the selected outgoing link Li. 

Having described various operations that may be 
^5 performed in accordance with the present invention, 
U exemplary apparatus, methods and data structures for 
Sl5 implementing such operations are now described in § 4.4 
below. 



§ 4.4 EXEMPLARY METHODS^ DATA STRUCTURES, AND 

APPARATUS FOR PERFORMING THE EXEMPLARY 



■220 OPERATIONS 

Exemplary methods and data structures for 
implementing various operations of the present invention 
are described in § 4.4.1. Then, exemplary apparatus for 
25 implementing various operations of the present invention 
are described in § 4.4.2. 

§ 4.4.1 EXEMPLARY METHODS AND DATA STRUCTURES 

30 Figure 3 is a high-level flow diagram of an 

exemplary method 210' for effecting various VOQ operations 
210. Referring to conditional branch point 310 and block 
32 0, if a cell has arrived at the VOQ but has not yet won 
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arbitration (the VOQ is "non-empty")/ a request is 
broadcast to all link arbiters (or link arbitration 
operations 23 0) of the input module (IM) 12 0 to which the 
VOQ belongs. (Recall, e.g., the first communication 215 of 
5 Figure 2 . ) 

Still referring to Figure 3, as indicated by 
conditional branch point 330 and block 340, if the cell (or 
the VOQ in general) won arbitration (e.g., all rounds of 
10 arbitration -- through to the central module 14 0) (Recall, 
e.g., the sixth communication 270 of Figure 2.), the cell 
is sent (e.g., in the upcoming cell time slot) . Although 
^ not shown in Figure 3, these steps may be continuously run. 
Mi. State information 212 of the VOQ may be updated at this 
j|;i5 point. Alternatively, blocks 32 0 and 34 0 may be triggered 
upon the occurrence of the relevant events. 

Figure 4 is a high-level flow diagram of an 
m exemplary method 23 0' for effecting a link arbitration 

operation 230. Referring to conditional branch point 410, 
it is determined whether or not a request or requests were 
received from a VOQ or VOQs . If not, the method 230' is 
left via RETURN node 440. If, on the other hand, such a 
request or requests is received, the rest of the method 
25 230' is effected. Thus, the receipt of request (s) from 
VOQ(s) may serve to trigger the main steps of the method 
230' . 

As indicated by block 420, the link arbitration 
30 method selects one VOQ from among those sending requests. 
(Recall, e.g., 230 and 235 of Figure 2.) The arbitration 
420 may be done based on the position of a pointer which is 
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updated in accordance with a round-robin discipline. The 
pointer may move through all of the VOQs . If the pointer 
currently points to a VOQ that did not send a request, it 
may move to a next VOQ, repeatedly, until it points to a 
5 VOQ that submitted a request. Referring back to Figure 4, 
the method 230' then sends a grant to the selected VOQ as 
indicated by block 43 0. (Recall, e.g., the second 
communication 237 of Figure 2.) The method 230' may then 
be left via RETURN node 440. Note that state information 
10 (e.g., the pointer) may be updated at this point, or, 
alternatively, may be updated later. 

B Figure 10 illustrates exemplary state information 

235' that may be used by the link arbitration operation 
JiaS 230'. As shown, a table 1010 may include a plurality of 

records, each record including a VOQ identifier 1012 and a 
O; field 1014 indicating whether or not a request was received 
Jif (in the current iteration) from the corresponding VOQ. A 
fB pointer 1016 may cycle through the records in a round-robin 
u20 manner. The VOQ identifiers 1012 may be ordered such that 
a given VOQ within various groups of VOQs are serviced 
first, before a next VOQ within the various groups of VOQs 
are serviced. As will be appreciated from the examples 
described in § 4 . 5 below, such an ordering of the VOQs 
25 hastens an advantageous desynchronization of pointers. 

The data structure 1020 may be used to indicate 
whether or not the outgoing link is reserved for a next 
cell time slot. If so, the link arbitration method 230' 
30 can ignore requests from VOQs (and/or inform the VOQs that 
it is taken) . The data structure 103 0 may be used to 
indicate whether or not a VOQ, which was selected, chose 
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the outgoing link in its own arbitration. Finally, data 
structure 1040 may be used to indicate whether or not a 
central module request was granted. 

5 Figure 5 is a high-level flow diagram of an 

exemplary method 24 0' for effecting a VOQ arbitration 
operation 24 0. Referring to conditional branch point 510, 
it is determined whether or not one or more grants are 
received from the link arbitration operations 230. 
10 (Recall, e.g., the second communication 237 of Figure 2.) 

If not, depending on the cell time slot and the time needed 
% for a dispatch determination (i.e., to match a VOQ with a 
ro CM), additional iterations (requests) may be possible, 
ii Accordingly, as indicated in conditional branch point 52 0 
tl5 and block 53 0, if there is time left for another iteration, 
H'^ the VOQ arbitration method 240' may inform a VOQ operation 
Oi 210 to rebroadcast a request. 



p; Referring back to conditional branch point 510, 

u20 if one or more grants are received, the method continues to 
block 540 where one of the candidate grants is selected. 
The selection 540 may be done based on the position of a 
pointer which is updated in accordance with a round-robin 
discipline. The pointer may move through all of the 
25 outgoing links 130. If the pointer currently points to an 
outgoing link 13 0 that did not send a grant, it may move to 
a next outgoing link 13 0, repeatedly, until it points to an 
outgoing link 130 that submitted a grant. Then, as shown 
in block 550, a grant is sent to the outgoing link 
30 operation associated with the selected one of the candidate 
grants. Although not shown, in an alternative method, the 
VOQ can submit a request, on behalf of the selected 
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outgoing link (Li) , to a central module arbitration. 
(Recall 290 of Figure 2.) The method 240' may then be left 
via RETURN node 560. 

5 Figure 11 illustrates exemplary state information 

245' that may be used by the VOQ arbitration operation 240. 
As shown, table 1110 may include a plurality of records, 
each record including an outgoing link (Li) identifier 1112 
and a field 1114 indicating whether or not a grant was 
10 received (in the current iteration) from a corresponding 
outgoing link. A pointer 1116 may cycle through the 
records in a round-robin manner. 

The data structure 112 0 may be used to indicate 
15 whether or not a cell is buffered at the VOQ. The data 
structure 113 0 may be used to indicate whether or not a 
grant (s) was received from an outgoing link(s) . If not, no 
VOQ arbitration operation 240 is needed. Finally, the data 
structure 114 0 may be used to indicate whether or not a 
20 central module grant was received (e.g., by a matched 
outgoing link (Li) ) . Such information may be used, for 
example, for updating the pointer 1116. 

Figure 6 is a high-level flow diagram of an 
25 exemplary method 250' for effecting an outgoing link 

operation 250. Referring to conditional branch point 610, 
it is determined whether or not a VOQ grant is received. 
(Recall, e.g., the third communication 247 of Figure 2.) 
If so, a request is sent to a central module arbiter, as 
30 indicated by block 52 0. (Recall, e.g., the fourth 
communication 255 of Figure 2.) Recall that in one 
alterative, the VOQ may submit such a request on behalf of 
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the outgoing link (Li) . 

Referring to conditional branch point 63 0, it is 
determined whether or not a grant is received from the 
5 central module arbiter. (Recall, e.g., the fifth 

communication 2 67 of Figure 2j If so, the VOQ (e.g., VOQ 
operation 210) matched with the outgoing link 130 is 
informed (so that it can send its head-of-line (HOL) cell. 
(Recall, e.g., the sixth communication 270 of Figure 2.) 

10 Although not shown in Figure 6, these steps may be 

continuously run. Alternatively, blocks 62 0 and 640 may be 

! triggered upon the occurrence of the relevant events. 



lV Figure 7 is a high-level flow diagram of an 

515 exemplary method 2 60' for effecting a central module 

arbitration operation 260. Referring to conditional branch 
p.i: point 710, it is determined whether or not one or more 
W requests were received from one or more outgoing links (Li) 
S 13 0. If not, the method 2 60' may be left via RETURN node 
H20 740. If, on the other hand, one or more requests were 

received from one or more outgoing links 130, the central 
module arbitration method 260' may select one link from 
among the candidate links that sent a request, as indicated 
by block 720. The selection 720 may be done based on the 
25 position of a pointer which is updated in accordance with a 
round-robin discipline. The pointer may move through all 
of the outgoing links 130. If the pointer currently points 
to an outgoing link 13 0 that did not send a request, it may 
move to a next outgoing link 13 0, repeatedly, until it 
30 points to an outgoing link 130 that submitted a request. 
Referring to block 73 0, the method 2 60' may then send a 
grant to the selected link. (Recall, e.g., the fifth 
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communication 267 of Figure 2.) The method 260' may then 
be left via RETURN node 740. 

Figure 12 illustrates exemplary state information 
5 2 65' that may be used by the central module arbitration 
operation 250. As shown, table 1210 may include a 
plurality of records, each record including an outgoing 
link (Li) identifier 1212 and a field indicating whether or 
not a request was received from the corresponding outgoing 
10 link (Li) . A pointer 1216 may cycle through the records in 
a round- robin manner. The data structure 122 0 may be used 
'i: to indicate whether or not an outgoing link (Li) was 
© selected in the previous or current cell time slot. 

+;|15 Note that if a request from a matched 

N= VOQ-outgoing link is not granted, such a request may be 

p,, resent to a central -module arbiter in a next cell time slot 

W (e.g., if pointers related to ungranted requests are not 

ff^ updated.) 

§ 4-4.2 EXEMPLARY APPARATUS 

Figure 8 is a high-level block diagram 
illustrating exemplary components and interconnections of 

25 an exemplary apparatus 800 that may be used to effect at 

least some of the operations of the present invention. An 
exemplary input module (IM) 12 0' may include virtual output 
queues (VOQs) 810, each of which includes (or more 
generally, is associated with a corresponding) an arbiter 

30 815, and link controllers 820, each of which includes (or 
more generally, is associated with a corresponding) an 
arbiter 825. An exemplary central module 140' may include 
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arbiters 830. The VOQ operations 210 may be effected on 
the virtual output queues 810. The VOQ arbitration 
operations 240 may be effected on the VOQ arbiters 815. 
The link operations 250 may be effected on the link 
5 controllers 820. The link arbitration operations may be 
effected on the link arbiters 825. Finally, the central 
module arbitration operations may be effected on the 
arbiters 830. Naturally, such operations may be physically 
located elsewhere. 

10 

Various signaling lines or links may be provided. 
S To simplify the drawing, signaling lines coupled with VOQs 
W 810b and 810c, or their associated arbiters 815b and 815c, 

are not shown. Signaling lines, depicted by solid lines 
S;i5 840, may be used to broadcast requests from a VOQ 810 to 
M= each of the link arbiters 82 5 in the input module (IM) 
p;; 120'. (Recall, e.g., the first communication 215 of Figure 
W 2.) Signaling lines, depicted by short dashed lines 850, 
03 may be used by each link controller arbiter 82 5 to send a 
]i0 grant to a selected VOQ arbiter 815. (Recall, e.g., the 
second communication 237 of Figure 2.) Signaling lines, 
depicted by dot -dash lines 860, may be used by a VOQ 
arbiter 815 to send a grant to a selected link controller 
820. (Recall, e.g., the third communication 247 of Figure 
25 2.) At this point, a VOQ 810, winning arbitration, may be 
matched with an outgoing link. 

Signaling lines, depicted by double dot -dash 
lines 870, may be used by the link controllers 820 to send 
30 requests to the arbiters 830 of the central modules 140'. 

Signaling lines, depicted by double dash-dot lines 880, may 
be used by the central module arbiters 83 0 to send a grant 
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to a selected link controller 820. Finally, the link 
controllers 82 0 may use signaling lines 8 90, depicted by 
spaced dotted lines, to inform the appropriate VOQs 810 
that they have won arbitration and can therefore send their 
5 head-of-line (HOL) cells (e.g., in the upcoming cell time 
slot) . 

The VOQs 810, their arbiters 815, the line 
controllers 820, their arbiters 820, and the arbiters 830 
10 of the central modules 14 0' may be effected by programmable 
logic arrays, application specific integrated circuits, 
'S! and/or microprocessors operating in accordance with stored 
03 instructions. Memory (referred to generally as a ''machine 
tl_ readable medium") may be used to store the various state 
*15 information (Recall, e.g. elements 212, 235, 245, and 265 

of Figure 2.) used by these components. Similarly, memory 
p. can be used to buffer cells at the virtual output queues 
if I 810. 

1^-^20 § 4.5 EXAMPLES ILLUSTRATING OPERATIONS PERFORMED 

BY AN EXEMPLARY EMBODIMENT 

Figures 9(a) through 9(g) illustrate an example 
of operations of the first stage of an exemplary 

25 dispatching method. In this example, the invention is 
embodied in a Clos-network switch, such as that 100 
illustrated in Figure 1. In the exemplary switch 100', 
n=m=k-2. To simply the drawings. Figures 9(a) 
through 9(g) do not show the second input module (IMd)), 

30 the second central module (CM(1)), or the second output 
module (0M(1)). As shown in Figure 9(a), to simplify the 
explanation, as well as to obtain desynchronization more 
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quickly, and in one embodiment of the output link 
arbitration method 230', the order of the VOQ(i,j,h) in 
IM(i) is redefined as VOQ(i,hk+j) as shown. Thus, in 
general, a pointer for use with the outgoing link 
5 arbitration method 230' and following a round-robin 
discipline, will cycle through the VOQs as follows: 

VOQ(i, 0, 0) ; 
VOQ(i,l,0) ; 

10 

VOQ(i,k-l,0) ; 

VOQ(i,0,l) ; 
W VOQ(i,l,l); 
M.. — ; 

|ri5 VOQ(i,k-l,l) ; 



VOQ (i, 0,n-l) ; 
E VOQ(i,l,n-l) ; 

vf\i ... ; 

H^20 VOQ(i,k-l,n-l) . 

In this way, the arbiter will cycle through a particular 
VOQ within various groups of VOQ first, and then through 
subsequent VOQs within each group. 

25 

In the following example, assume that VOQ(0,0), 
VOQ (0,3), VOQ (0,4), and VOQ (0,6) are non-empty. As shown 
in Figure 3 (b) , these non-empty VOQs each broadcast a 
request to all link arbiters in their input module (IM(0)). 
30 (Recall, e.g., the first communication 215 of Figure 2, as 
well as 310 and 320 of Figure 3.) 
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As shown in Figure 9(c), it is assumed that 
arbiters associated with outgoing links Li(0,0), Li(0,l) and 
Li(0,2) prefer VOQ( 0,0) , (VOQ(0,0) andVOQ(0,l), 
respectively. Since VOQ(0,0) is among those VOQs to 
5 broadcast a request, the arbiters associated with outgoing 
links Li (0,0) and Li (0,1) each send a grant signal back to 
VOQ{0,0) . On the other hand, since VOQ(0,1) was empty and 
did not broadcast a request, the arbiter associated with 
outgoing link Li (0,2) will try subsequent VOQs until one 
10 that sent a request (i.e., a non-empty VOQ) is encountered. 
In this case, the next VOQ that sent a request is VOQ(0,3). 
S Accordingly, as shown in Figure 9(c), the arbiter 
5 associated with outgoing link Li (0,2) sends a grant signal 
U. back to VOQ(0,3). (Recall, e.g., the second communication 
;fcl5 237 of Figure 2, as well as the method 230' of Figure 4.) 

Q Referring to both Figures 9(c) and 9(d), it is 

iff assumed that arbiters associated with virtual output queues 
m VOQ(0,0) and VOQ(0,3) both prefer outgoing link Li(0,0). 
5^20 Since both outgoing link Li (0,0) and Li (0,1) broadcast a 
grant to VOQ(0,0), the arbiter associated with VOQ(0,0) 
must select one. In this example, it selects outgoing link 
Li (0,0) and sends a grant signal back as shown in Figure 
9(d). On the other hand, since only the outgoing link 
25 Li (0,2) sent a grant to VOQ (0,3), the arbiter associated 
with VOQ (0,3) will try subsequent outgoing links until one 
that sent a grant is encountered. In this case, the next 
(and indeed the only) outgoing link to send a grant is 
Li(0,2). Accordingly, as shown in Figure 9(d), the arbiter 
30 associated with VOQ (0,3) sends a grant signal back to 
outgoing link Li(0,2). (Recall, e.g., the third 
communication 247 of Figure 2, as well as 510, 540 and 550 
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of Figure 5 . ) 

Assuming that more iterations of the foregoing 
steps are possible within one cell time slot (or more in 
5 certain embodiments), notice that VOQ(0,4) and VOQ(0,6) are 
non-empty, but were not matched with an outgoing link 
during the first iteration. Accordingly, as shown in 
Figure 9(e), these VOQs may rebroadcast their requests. 
(Recall, e.g., 52 0 and 53 0 of Figure 5.) Note that if the 
10 outgoing link arbiters maintain a certain information, such 
a rebroadcast would not be necessary. The outgoing link 
X Li(0,l) is the only non-committed link. Based on its 
5 pointer, the arbiter associated with the outgoing link 
P Li(0,l) will prefer VOQ(0,0). However, since VOQ(0,0) was 
:5l5 already matched with an outgoing link in the previous 
H= iteration, and therefore did not rebroadcast a request, the 
fi: arbiter associated with outgoing link Li (0,1) will try 
;2 subsequent VOQs until one that rebroadcast a request is 
ij encountered. In this case, the next VOQ that rebroadcast a 
lilO request is VOQ (0,4). As shown in Figure 9(f), the arbiter 

associated with the outgoing link Li (0,1) sends a grant back 
to VOQ(0,4). Finally, as shown in Figure 9(g), the arbiter 
associated with VOQ (0,4) sends a grant back to the outgoing 
link Li(0, 1) . 

25 

Although not shown in Figures 9(a) through 9(g), 
each outgoing link that was matched with a non-empty VOQ 
during the first phase of the exemplary cell dispatch 
scheduling invention will request a central module. 
30 Arbiters at the central modules will arbitrate such 

requests. Once an outgoing link Lq of a central module is 
matched with an outgoing link Li of an input module (and 
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therefore to a VOQ) , the state information (e.g., pointers) 

for each of the arbiters may be updated and the cells may 

be dispatched from those VOQs matched with an outgoing link 

Lq . 

5 

§ 4.6 CONCLUSIONS 

The disclosed cell dispatch scheduling invention 
does not require buffers in the second stage, thereby 
10 advantageously avoiding cell out-of -sequence problems. 

Further, unlike random dispatching schemes having limited 
(e.g., 75%) throughput unless internal bandwidth is 
m expanded (e.g., speed-up - 1,582 is applied), simulations 
11 have shown that the cell dispatch scheduling invention can 
'H5 achieve 100 % throughput (independent of the number of 

z^. ;;- 

hh iterations of the first phase) under uniform traffic. More 
™ specifically, a VOQ that fails to win contention has to 
I"' store backlogged cells. Under uniform traffic, every VOQ 
m keeps such backlogged cells until the idle state (i.e., the 
F%0 state in which the internal link is not fully utilized) is 
eliminated that is, until the stable state (See, e.g., 
the article, N. McKeown, A. Mekkittikul , V. Anantharam, and 
J. Walrand, ^'Achieving 100% Throughput in an Input-Queued 
Switch,'' IEEE Trans, on Communications , Vol. 47, No. 8, pp. 
25 1260-12G7 (Aug. 1999) . This article is incorporated herein 
by reference) is reached. Once in the stable state, every 
VOQ is occupied with backlogged cells. In such a state, 
the arbiters (e.g., round-robin pointers) become 
desynchronized. Consequently, even when the offered 
30 traffic load is full, no contention occurs in the stable 
state. Therefore, the dispatch invention achieves 100 % 
throughput under uniform traffic. 
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Such 100 % throughput under uniform traffic 
occurs independent of the number of iterations. However, 
as the number of iterations (e.g., possible per cell time 
5 slot) increases, the delay performance becomes better. 
This is because the matching between VOQs and outgoing 
links Li within an input module (IM) will improve. When the 
offered traffic is low, the desynchronization of the 
arbiters (e.g., round-robin pointers) is less likely 
10 achieved. In such a case, using less iterations negatively 
impacts performance. This is because the matching between 
VOQs and outgoing links Li within an input module (IM) will 
worsen. 

15 Even under bursty traffic, the dispatch invention 

provides 10 0 % throughput, which is also independent of the 
number of iterations of the first stage. However, the 
delay performance of the bursty traffic is worse than that 
of more uniform traffic under heavy load conditions. 

20 

The throughput of the cell dispatch scheduling 
invention is better than that of random dispatching even 
when traffic is unbalanced (i.e., not uniform). Further, 
the fact that the cell dispatch scheduling invention can 
25 use round robin arbiters ensures fairness under non-uniform 
traffic conditions , 

The cell dispatch scheduling invention has a time 
complexity O(log nk) for each iteration of the first phase. 
30 Therefore, if there are m iterations (such that outgoing 
links are matched with the VOQs in an IM) , the time 
complexity of the first phase is 0(m log nk) . The second 
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phase has a time complexity of O(log k) . Therefore, the 
time complexity of the cell dispatch scheduling invention 
is approximately O (m log nk) = O (m log N) , where N is the 
number of ports. If the number of iterations of the first 
phase is set to i, where 1 < i < m, the time complexity is 
expressed as 0(i log N) . Given this time complexity, the 
cell dispatch scheduling invention is scalable and may be 
used successfully in large scale switches. 
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